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ABSTRACT 

The University of Iowa has been involved with the 
development^ implementation^ and operation of computer-based 
test-item pools and a test construction and course management system 
titled IPSIM (Iowa*s Item Pool System for Instructional Management)^ 
originally devised for a freshman medical course in the Pathology 
Department of the College of Medicine. The IPSIH system provides the 
user with the ability to: (1) create and manage item pools; (2) 
produce "paper-and-pencil" tests^ including computer-generated forms; 
(3) provide Coursewriter III code for interactive test 
administration; (4) construct multiple-choice questions from a 
prototype statement; (5) score tests; (6) analyze student performance 
data; and (7) compose student progress reports. The system appears to 
have wide ranging potential for other subject matter areas. 
(Author/KKC) 
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INTPODUCTION 

Throughou-: *he past twen*:y-four months The University of Iowa has 
b^en involved wi-h development^ implementation^ and operation of 
computer based test item pools and a test-construction and course- 
management system titled IPSIM (Icwa«s Item Pool System for 
Instructional Management). The evolution of this system has been 
influential in convincing us that test item pools are a valuable 
resource for educators and that course-and pool-management programs 
ar*^^ an effective utilization of the computer in the instructional 
process, particularly beneficial for self-paced, individualized 
courses • 

This type of computer application to the instructional process 
appears -^o possess wide ranging potential and to be viewed by moct 
users ^classroom instructors) as a needed, cost-effective investment 
cf time, effort, and money. The computer aids the instructor in 
several cf the most time-consuming and least-valued parts of his/her 
job—the production and grading of tests and the analyses of their 
r^^£ul-s, "-he evaluation of studen^ performance, and the keeping of 
class records. Additionally, information concerning class performance 
is provided to assist instructors in identifying and correcting weak 
parts of the curriculum and in adjusting instruction to the 
performance and needs of *he students. 

The IPSIM system, originally devised for a freshman medical 
course in the Pathology Department of the College of Medicine, is 
iippl^ment-a on The University of lowa^s IBM 360 Model 6S computer. 
The Pathology it-m pool evolved from the efforts of members of a 
'nedical educators* group called GRIPE ( Group for Research and 
Instruction in Pathology Education) and is ccniprised of ever 4,000 
qu*='Sttonsj and student-performance data, with 30 medical schools in th<=^ 
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Uni-red Stages and Canada pres-ently utilizin9 its capabilities. The 
basic system has been extended to other academic areas at The 
Universi-y of Iowa through th*? development of item pools in Materials 
Science for the College of Engineering, in Introductory Chemistry for 
the Chemistry Department, and through work on an item pool for the 
Division of Educational Media. 

A detailed description of the original system developed to manage 
-'le medical i^em pool and the self-paced Parhology course is contained 
in a paper presented at the Fifth Conference on Computers in the 
Ordergraduate Curricula held in Pullman, Washington, June 24-26, 1974. 
The following section will briefly summarize the features of this 
system with the remainder of the paper detailing the significant 
enhancements made to the IPSIM system for the application in 
Chemistry. 
FEVIEW OF IPSIM 

The IPS"^M system which consists of approximately twenty computer 
proqrams wri-ten in PL/1 (Programming Language One) is totally 
operational, and we are fully utilizing its capabilities. The item 
pools, ^.he core of the system, are stored as keyed indexed sequential 
ftl<^s with variable length records utilizing IBM«s PL/1 data base 
managemen-^, routines and disk storage conventions. An item pool is 
u-ually comprised of tes^t questions as well as other descriptive 
information (-.g. , k'^^y phrases or words, cross references, statistical 
characteristics, e*'-c.) which are filed by the system. 

In computerizing a pool of questions, a classification scheme for 
organizing an'l indexing items must be developed. Sven ^.houqh the 
syst»=^m will also permit alphabetic letters -to be used if dei^ired, an 
eiqht-aigit number is most often employed for cataloguing and 
Q re-rieving it-ms in our work at The Hniv^rsi^y of Iowa. The first two 
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digits of an item pool number are utilized to designate the MCA 
(ma jor-content area) and the following two digits denote 
subcategories, the TD«s (topic descriptors) , under each HCA, The 
remaining four digits represent the sequence number of rhe item (three 
digits) and the version number (final digit) • 

Employing the generic key capability of file management, items 
c^n be selected by specifying either the entire number or any generic 
part of the HCA-TD number. Illustrating the generic concept through 
use of alphabetic letters for simplicity, all names in a telephone 
book file beginning with the letter A would be retrieved by calling 
for A, all names beginning with Al by AL, names with the first three 
letters of Alb by ALB, all names having the first four letters Albe by 
ALBS, etc* Or an entire specific name such as Albert would be 
retrieved by specifying ALBERT* Since most item retrieval is either 
for a relativ=*ly small number of specific entries or for a large 
number of entries from a generic subset, the generic key capability 
has been extrfmely useful. 

In addition to the item number, the structure of a typical item 
pool entry may include the following components: (1) the question 
and/cr textual material; (2) the correct answer; (3) the question 
format, a number used by the system for questions having a common set 
of attributes; (4) the MCA and TD phrases; (5) the SCA (secondary- 
conten- area) which indicates a cross reference; (6) the KP(key 
phrase) , a description cr reference of the knowledge required by a 
student to correctly answer the question; and (7) the performance 
data, wi*-h one line of data per each occasion the item is used^ 
Comprising the performance da*a is information such as the percentage 
of students selecting each response (foil) or failing to indicate a 
O choice cf answers; the number of students tested; the MPL (minimum- 

ERIC ^ 



HcClain Page 4 

pass lev-:^!) ; the aiscrimination index; and the difficulty of a 
question. 

To create and manage the it^m pools, computer programs have been 
written to bank (add questions or other materials/information to a 
pool), delete, edit (correct errors), update (add performance data), 
and rename (change classification number) the test questions. An item 
can be banked which contains any number or all of the components 
listed in the preceding paragraph, a feature enabling the IPSIM system 
to be flexible and thus permitting it to be utilized in creating 
various pools (e.g., test questions, course objectives, text 
references, etc.). The sequence and version number of an item can 
either be supplied by the user upon banking the question or be 
automatically added sequentially to -^he MCA-TD number by the system. 

The system provides three basic forms of output—an item listing, 
an index, and a test. A listing is simply a printed copy of an item 
con-'-air.ing one or any number of the seven components previously 
discussed. \n index, a catalog with each entry a brief description of 
the item, provides the user with a quick method of identifying 
appropriate questions for the particular application. A test, being 
defined as a series of questions regardless of its use or purpose, can 
be generated in two ways: a printed copy for reproduction as a 
"paper-and-pencil " test or a series of questions coded in CWIII 
(Coursewriter 111), IBH's author language for interactive 
administra-^ion by the computer via a terminal. 

Regardless of how a test is administered, a very important and 
of*en omitted follow-up procedure is the collection of performance 
data. The IPSIH system is programmed to utilize input from either 
nark scan cards or CWIII student records to perform an item and tes- 
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analysis. The student-performance data obtained is added to thf- pool, 
the update making it more informative, 

Tc accommodate self-paced, individualized courses com,puter 
programs have been devised to record student data and provide status 
reports detailing student progress on individual units. These reports 
can b^ run at spvcified p^^riois of time and are valuable assets to 
course management • 

In summary, the IPSIH system is designed to: (1) create and 
manege item pools; (2) generate tests; (3) score tests; (4) analyze 
studen-^-perf ormance data; and (5) provide studen + -progress reports, 
CROSS REFERENCING 

It has been our experience at The University of Iowa that 
selection of a classification scheme is by far the single most- 
important aspect in building any item pool. The classification scheme 
has probably failed, if the user can ever make a simple request for 
information from a pool and receive a major percentage of the pool's 
entries in response. The scheme needs tc act as a very definite 
filtering process, 

A dominant philosophy in our work has been that if a specific 
item does indeed fit into more than one category of the classification 
scheme, either the scheme has failed or the item should be handled as 
two absolutely distinct items* Unfortunately, the multiple-entry 
principle has proven to be not entirely satisfactory for all 
applications. One solution is to allow liberal cross referencing 
within a pool. However, there is potential danger here in tha*^ the 
des-ired filtering process can easily be negated by indiscrimina*:e 
cross referen'^ing • 

Our solution a-^ The University of Iowa, which we term th^ 
"parent/alias concept" and advise our UFers to use with cau-^ricn. 
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involv*=iS one-way referencing. The parent is comprised of the question 
or -^extr any of the descriptors, and any performance data. 
Furthermore, it includes a count of the aliases which reference the 
p^^rrnt (used -o assure that the parent is not deleted or renamed 
before all aliases are removed from the pool) . Each alias contains 
only a reference to the parent, the parentis item number. There is nc 
doub": but that this method can be abused, but we believe it provides 
just enough flexibility so as to handle the exceptions in item 
classification without negating the filtering prccess imposed by the 
gen-ric classification scheme. In cur opinion, cross referencing is 
definitely not an alternative to a classification scheme. 
COMPUTER GENERATED MULTIPLE CHOICE QUESTIONS 

To be of greatest benefit and value for a self-paced course an 
item pool should be large enough so there is no need for test 
security. In fact, the pool of questions might be an important 
learning aid for -he student if it were to be made available tc 
him/her. Bu- with static multiple-choice questions, probably more 
tnan 2,000 it-ms for a one-semester course would be needed. It is 
believed, however, that a computer-generated question could be 
equival'=^nt +o a large number of static questions in the item pool, 
depending upon the creativity of ^he writer. 

The IPSIM systpm was expanded to not only be used as an aid in 
preoarinq t^sts from a pool of items but also be employed as an aid 
in producing the -^est i~em itself. Even "rhough it is most effective 
for developing questions wivh numerical answers, computer-question 
generation can also be utilized for devising non-numerical questions. 

Th^ system retrieves a skeleton or prototype of an item from the 
pool in a manner identical to ^hat for regular test-question 
^ selection. The prototype states the rules for creating the specific 
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variable componerts in both a gu?<t:.on and its multiplG-choice foils. 
The rules specify exactly how -^o genera'^'e entries for the variable 
parts of it<=^m3: the type of entry (alphabetic, decimal, or integer) , 
the range of numeric values, the number of significant digits, and the 
algorithms for selecting or calculating variables. Only this 
pro-^otyp- is banked in the item pool, with the system constructing the 
actual question during test production. 

The effectiveness of computer-generated questions can best be 
illustrated through use of a simple example. The instructor writes 
the item prototype as follows: 

What volume of »R 3 .1 .5' M. «T HCL; HBr; HI; HC104;' 

will neutralize 10 40^ mis. of »R 3 .1 .5" 

'T NaOH; KOH; NH3; •? 

(A) 'RA 3 X3*xa/Xl« mis. or »TA X2 » 

(B) »PA 3 2*X6» mis. of 'TA X2» 

(C) ^RA 3 X6/2> mis. of • TA X2 » 

(D) 'FA 3 1/X6^ mis. of »TA X2» 

(E) »PA 3 X3^Xl/Xa« mis. of 'TA X2» 

Enclosed within each se": of single quotation marks is a rule for 
creating the variable and is replaced by the computer-generated entry 
during ^est production. Immeiiately following the first quote mark is 
one of the three characters which informs the computer of the type of 
"ex* *-o be created: an P for real decimal numbers, an N for integer 
numbers, and a T for t^x*:ual (alphabetic) materials. Next is either a 
blank space for randomly generated variables or the letter A for tex- 
generated by an algorithm. Thus, the six forms of computer-generated 
variables are R, T, RA, NA, and TA. 

In -:he above example th^ first variable it<=^:n, denoted by the 
Q syst^^m as Xl, is a randomly generated decimal number of thre- 
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significant digits havii:g a numeric value between .1 and .5 in 
magni-ude. The second item to be generated, 12, signifies randomly 
chosen textual material from a given list. X3 is a random integer 
b-stweer 10 and ao. The computer-generated variable for X4 and X5 are 
similar in form ^o XI and X2, respectively. 

In the first f oi] , the variable X6 is a decimal number with three 
significant digits calculated by the given algorithm X3*Xil/Xl. It is 
important to note that the algorithm (formula) can utilize any of the 
arithmetic operations as well as some basic functions {e.g., SQE, LOG, 
SIN, and COS). Furthermore, the algorithm can reference any variable 
in '*:he item preceding or following it. The second text of the A foil 
is a simple reference to the textual material selected in X2. 

When the generated multiple-choice question is printed as part of 
an examination, the order of the foils is scrambled. A sample 
execu*:ion of the aforementioned item with the foils unscrambled is: 

What volume of .217 H. HBr will neutralize 26 mis. of .346 «• NaOH? 

(A) ^1.5 mis. of HBr 

(B) 82.9 mis. of HBr 

(C) 20.7 mis. of HBr 

(D) .0241 mis. of HBr 
(F) 16.3 mis. of HBr 

The computer-generated question system was designed so that: (1) 
questions can be constructed by instructors who have a minimal 
knowledg<= of computer programming, and (2) as -^he prototypes are 
written in th? qu-stior. format accepted by IPSIM, computer-generated 
itc-m? can be mixed with and selected in the same manner as the static 
entries. 
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AUTOHATIC-TESr GENERATION 

For its self-pace a courses^ The University of lowa^s 
undergraduate chemistry division had a need for generating a large 
number of unique tests over specific subject-matter areas. To satisfy 
This need and as a response to the increasing interest in automatic- 
test generation, IPSIH was augmented to perform automatic-test- 
generation in a manner which has proven to be quite successful. 

According -^-o directions given by the user, test generation 
typically proceeds as follows: Construct three forms of a test which 
will b^ of average difficulty (difficulty index of the test about 60) 
but will result in a wide range of scores (variance of the difficulty 
index about 600) . 

To satisfy the request, IPSIH performs two-phase selection 
process : 

(1) In Phase I, all applicable items are grouped. Then for each 
form of a test, a random subset of a sufficient size to perform 
the next phase is selected. 

(2) In Phase II, "rhe items from each subset which best fit the 
user's "^-est specification are chosen. Once the items are 
determined, the user can employ any of the test printing, 
automatic scoring, and analysis capabilities available wi-.h 
IPSIM. 

This general description does not serve to demonstrate any specifics 
of IPSIH's automatic- test generation. In par-^.icular , what are 
"applicable it^^ms" in Phase I? In Phase II, what constitutes a "bes^. 
fit"? 

A best fi*: is achieved by weighting each item in a given subset 
by -^-h^ weighting formulas. Items with the highest weight will 
Q constitute thr set of items which most closely approximate the test's 
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desired characteristics* Elemen-s involv'rd in the weighting scheme 
are summarized as follows: 

Subset 
calculated avg. 



Difficul-^-y Index 



Variance of Above 



calculated 



Discrimination Index ca3cul.^*ted avg. 



Past Usage 

No* of Items in a 

"category" 

How badly user wants 
items in a "category" 



calculated 
NA 



It em 
actual/default 
(can override) 
NA 

actual/def a'olt 
(can override) 
actual 
NA 

default 

(can override) 



Desired 
default 
(can override) 
Default 
(can override) 
NA 

NA 
NA 

NA 



In addi'-ion to the six elements in the above table, four other 
relevant parameters which the user can override are incorporated to 
allow him to experiment with and/or massage the weighting fcrmulas. 

Applicable items are ones which meet the user's specifications 
for inclusion. For example, DEFDISC (4 0) , TEST (CATEG (#22-33) ) is a 
complete test specification. The applicable items are all those found 
under the HCA-TD of #22-33. IPSI« utilizes its default values for 
everything except the descrimination index value indicated by the 
DEFDISC which is to be assigned to any items lacking such data. 

Two oth-r parameters which can be overriden are the number of 
forms (d-=^fault = 1) and the number of questions in the test (default = 
25) . Thus, the specification tc get four, thirty-five i-^em forms of 
the above test is: DEFDISC (40), TEST (NOFOPM (4) , NOQUEST (35) , 
CATEG (#22-33) ) • Modifying the above example to include items from 
^ category #88-88, i^ becomes DEFDISC (40), lEST (NOFORM (4) , NOQUEST (35), 
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CATEG(UNION(#22-33, #88-88) )) , where UNION is ^.he normal set union 
operation. 

One of the primary purposes of the CATEG (category) descriptor is 
to equalize any disparity in the number of items frcm a specific HCA- 
TD, If #22-33 has significantly fewer items than #88-88, the 
probability of final inclusion of items from #22-33 is a good deal 
less than that of items from #88-88. In some applications this is 
desirable but in others it is not. The following example demonstrates 
how to equalize these probabilities: DEFDISC(40), 
TEST(NOFORH (4) ,NOQaEST(35) , CATEG (#22-33) , CAT£:G (#88- 88) ) . 

Thus far we have introduced the basic form of IPSIH^s test- 
specification language and have overriden several of the test- 
description parameters. However, the most extensive aspect of the 
language is the capability of specifying criterion for the inclusion 
of items. In the preceding example, the criterion was merely all 
items cf #22-33 and all of #38-88. 

Adding— include items from #22-33 which are not aliases for 
entries in #88-88--the test specification becomes: 

DEFDISC(40) ,TSST (NOFORH (4) ,NOQUEST(35) , CATEG (UNION ( QUAL (NOT (ALIAS 

(#88-88)) ,#22-33! ,#88-88)) . This introduces the QUAL descriptor which 
stands for qualification. (Items from #22-33 are qualified to no+ be 
aliases of it^ms in #88-88.) Also, the example introduces one cf the 
three logical operators; AND, OP are the other two. ALIAS is a so- 
called bottom-level descriptor because it is applied directly to the 
items. If, instead of specifying items which are not aliases for 
entries in #88-88, we specify items which are either computer- 
generat^rd questions (denoted by a FORMAT of 9) or questions which are 
aliases, *-he specification changes to 
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DEFDISC (40) ,TSST(NOPORH(a) ,NOQUEST(35) , CATEG (UNION (QUAL (OR (FORMAT (9) , 
ALIAS) ,#22-33) , #88-88) ))• 

It is not difficult to ses hew complex a test specification can 
become. On the ether hani, suppose the user says he wants items from 
#22-:i»3 and #88-88* These items should have a difficulty cf at least 
fifty and be computer-generated questions or aliases. First of all, 
the three qualifications can be logically combined in several 
different ways. Secondly, which parts of the qualifications are, 
intended to apply to #22-33? Which to #88-88? Which to both #22-33 
and #88-88? This illustrates the age-old dilemma cf the user 
translating what he think? he needs to tell 'the computer into what the 
computer needs to be told. 

A^ this point in automatic-test generation we feel it is 
important to have a language which can never be ambiguous to the 
computer. Our test-specification language is powerful enough to 
express any possible inclusion criterion in an unambiguous manner and 
yet De as reasonable to work with as some computer programming 
languages. To illustrate both its heavy recursiveness and its 
flexibility in adapting to the user^s way of thinking, suppose two 
users ask for the following tests: 

All items from #22-33 and #88-88 I^ems from #22-33 which 

wi^h difficulty of a* least 50. have difficulty of at least 

But those from #22-33 mu?t be 50 and be either ccmputer- 

either computer-genera tei generated questions or be 

questions or aliases. aliases. Also, items from 

#88-88 which have difficulty 



of a^ least 50. 



The two test specifications are: 



^ CATEG (QUAL(GE(DIF,50) , CATEG (UNION (QUAL (AND (GE(DIF,50) , 
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UNION (#88-88, QUAL (CR (FORMAT (9) , 



0R(F0RMAT(9) .ALIAS)) ,#22-33) , 



ALIAS) ,#22-33) ) ) ) 



QUAL(GE(DIF,50) ,#88-88) ) ) • 



The first specification shows how recursion can be employed, with the 
en-ire category qualified to have a difficulty of at 3cdSt fifty* The 
items from #22-33 are then further qualified* In the second 
specification, each HCA-TD is independently qua,lified. Though the 
first specification is shorter, it is exactly the same specification 
as the second one* In fact, the judgment as to which is the more 
accurate depends upon the user's reasoning. 

In conclusion, automatic-test generation is achieved at The 
University of Iowa by a two-phase selection scheme. In the first 
phase IPSIM groups all applicable items, ones which meet the inclusion 
criterion specified by the user, to be considered for selection in the 
second phase. The criteria are expressed via a language which: (1) is 
powerful enough to express any inclusion criterion; (2) is as 
reasonable to work with as some computer languages; (3) is extendable 
in *:he sense that the addition of bottom-level descriptors have been 
anticipated and space allocated for them; (4) is flexible to the 
user's way of thinking. From the subset of randomly chosen items 
formed in Phase I, Phase II selects the se^ of items which best fit 
the characteristics desired for the test. Random selection provides 
unique subsets for each form of a desired test. Fitting of items -^o 
the desired t^st crit-^^rion is achieved by the weighting formulas. In 
addition to the desired characteristics of a test, all the parameters 
in ':hese formulas can be overridden by the user. 

IPSIM's automatic test-generation system has initially performed 
be^-^er than we had anticipated. Due to the flexibility in massaging 
^.he fitting criterion, th<= existence of varied item descriptors in th<^ 
item pools, and the complet en-^ss of the inclusion criterion aspect of 
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the language, we believ^ that this system has even greater potential 
than has yet been u-:ilizea. 
THE CHEMISTRY APPLICATION 

An application utilizing the enhancements of the IPSIM system has 
been th'^^- development of a self-paced introductory course in chemistry. 
A modified Keller plan for chemical principles was introduced on a 
trial basis Fall semester, 1974, in three first-year chemistry 
cours^^s, with a test group of 200 subjects cut of a chemistry student 
population of 1,000. General use of the plan was scheduled to begin 
Fall semester, 1975; but because of the success of the initial effort, 
two courses wi^„h a student enrollment numbering approximately 600 
employed this plan Spring semester, 1975, with the assis-^ance of the 
IPSIM system. 

In the modified Keller plan, the contents of each of the two 
spring semester chemistry courses is divided into six or more units. 
A student is permitted to take an examination over a unit as many 
times as and wherever he/she wishes. It is beli<^ved to be .re 
important for a student to acquire a strong general knowledge of 
chemical principles than to acquire mastery of a few topics. 
Therefore, the simpler mastery system of the Keller plan is not used as 
th^ standard for grading; instead, a student^s evaluation is 
determined by totaling the highest scores earned for each of the unit 
examinations. 

Thrcughoa-^ the course, students are aid^d by a tex*- , 
supplementary autotutorial materials, conventional lectures, 
discussions, and op^n tutorial sessions. It is anticipated that in 
fu-ur* years students will also be assisted through use of audiovisual 
mat-^^rials ani computer-assisted instruction techniques. The ultimate 
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goal is to provide the student with a truly individualized program 
while maintaining a high standard of quality* 

T+ should be clear that implementation of a self-paced program in 
an introductory course of nearly 1,000 students is impossible without 
the effec ivr utilization of technological innovations. Using the 
IPSIK system, we have been able to generate, score, and record results 
frcm the more than 200 different examinations required in the program. 
An ess<=intial feature has been the ability to generate multiple 
examinaticns of equal difficulty. 
SUHMAPY 

Briefly, the IPSIH system provides the user with the ability to: 
(1) creat<=T and manage item pools; (2) produce "paper-and-pencil" 
tests, including computer-generated forms; <3) provide CWIII code for 
interactive -est administration; (4) construct multiple-choice 
questions from a prototype statement; (5) score tests; (6) analyze 
studen-^-performance data; and (7) compose student-progress reports. 
The -^hree enhancements to the system (cross referencing, computer- 
generated mul-^iple-choice questions, and automatic test generation) 
were added to provide capabilities for the chemistry application, but 
they app'^ar to have wide ranging potential for ether subject-matter 
areas as well. With the additions to the original system, we have a 
flexible system which has proven to be a valuable asset to its users 
in a variety of instructional applications. 
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